Parallel video encoding

ABSTRACT

An electronic device includes a slice splitter and a slice encoder. The slide splitter is configured to split video data into a plurality of slices. Each slice contains a plurality of data blocks and each data block contains a plurality of data points. The slice encoder includes one or more video encoding circuits and is configured to encode the data blocks in a plurality of data streams concurrently to obtain encoded data streams and combine the encoded data streams into a combined data stream.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of International Application No.PCT/CN2016/093279, filed on Aug. 4, 2016, the entire contents of whichare incorporated herein by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF THE DISCLOSURE

The disclosed embodiments relate generally to data process, includingbut not exclusively, to video encoding.

BACKGROUND

Multimedia communication is a cornerstone of modern day commerce. Inmultimedia communication, video streams are transmitted from a sourcelocation to a destination through a communication network. A raw videostream often contains 15 to 60 frames per second, and each frameincludes hundreds of thousands or millions of pixels (also called pels,dots, or picture elements). Thus, storing and transmission the raw videostream would take large amount of storage space and transmissionbandwidth.

To enhance transmission efficiency, video encoding is often used tocompress video streams. At the source location, a video stream may beencoded to remove data redundancy, where the encoded data may be decodedwithout any loss of information at the destination (the techniques areoften referred to lossless compression/encoding). Additionally or inalternative, at the source location, the video stream may be encoded toremove data that loss of which has little impact to someone perceivingthe decoded video stream at the destination location (the techniques areoften referred to lossy compression/encoding).

While video encoding can be done in software, for multimediacommunication that has time delay requirements, video encoding is oftendone in hardware. Traditionally, video encoding is done by stationaryhardware such as encoding server/computer. The stationary hardware canprovide high quality compression in various protocols. However,stationary hardware is often bulky and power hungry, thus not suitablefor a mobile environment and that is the general area that embodimentsof the disclosure are intended to address.

SUMMARY

Described herein are systems, methods, storage media, and computerprograms that support encoding of video data. In one embodiment, anelectronic device is disclosed. The electronic device comprises a slicesplitter and a slice encoder implemented using one or more videoencoding circuit. The slice splitter is configured to split video datainto a plurality of slices, where each slice contains a plurality ofdata blocks, each data block containing a plurality of data points,which are processed in a plurality of data streams. The slice encoder,implemented using one or more video encoding circuits, is configured toencode the data blocks in the plurality of data streams concurrently andcombine the encoded data into a combined data stream.

The embodiments of the present disclosure provide video encodingcircuits that encode video data within a slice concurrently using aplurality of data streams, the concurrency of intra-slice encodingallows the video data to be encoded in real-time for multimediacommunications.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an exemplary illustration of a video communicationenvironment, in accordance with various embodiments of the presentdisclosure.

FIG. 2 is an exemplary electronic device including a video encoderaccording to one embodiment of the disclosure.

FIG. 3 illustrates video encoding processes according to one embodimentof the disclosure.

FIG. 4A illustrates data blocks being processed through encodingaccording to one embodiment of the disclosure.

FIG. 4B illustrates data blocks being processed through encodingaccording to another embodiment of the disclosure.

FIG. 4C illustrates concurrent encoding matching quantization accordingto one embodiment of the disclosure.

FIG. 5 is a flow diagram illustrating the intra-slice encoding accordingto one embodiment of the disclosure.

FIG. 6 is a flow diagram illustrating encoding operations in moredetails according to one embodiment of the disclosure.

FIG. 7 illustrates a post-encoding process according to one embodimentof the disclosure.

FIG. 8 illustrates composition of slice header information according toone embodiment of the disclosure.

FIG. 9 illustrates a slice index table according to one embodiment ofthe disclosure.

FIG. 10 is a flow diagram illustrating transmitting data to an externalstorage according to one embodiment of the disclosure.

FIG. 11 is a flow diagram illustrating storing data in a video encodingbuffer according to one embodiment of the disclosure.

FIG. 12 is a flow diagram illustrating dummy byte insertion intransmission of slice information to an external storage according toone embodiment of the disclosure.

FIG. 13 illustrates memory allocation of an external storage accordingto one embodiment of the disclosure.

FIG. 14 is an exemplary illustration of an electronic device forencoding video, in accordance with various embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The disclosure is illustrated, by way of example and not by way oflimitation, in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that referencesto “an” or “one” or “some” embodiment(s) in this disclosure are notnecessarily to the same embodiment, and such references mean at leastone.

In figures, Bracketed text and blocks with dashed borders (e.g., largedashes, small dashes, dot-dash, and dots) may be used herein toillustrate optional operations that add additional features toembodiments of the disclosure. However, such notation should not betaken to mean that these are the only options or optional operations,and/or that blocks with solid borders are not optional in certainembodiments of the disclosure. Also in figures, reference numbers areused to refer to various element or components, the same referencenumbers in different figures indicate the elements or components havingthe same or similar functionalities.

In the following description and claims, the terms “coupled” and“connected,” along with their derivatives, may be used. It should beunderstood that these terms are not intended as synonyms for each other.“Coupled” is used to indicate that two or more elements, which may ormay not be in direct physical or electrical contact with each other,co-operate or interact with each other. “Connected” is used to indicatethe establishment of communication between two or more elements that arecoupled with each other. A “set,” as used herein refers to any positivewhole number of items including one item.

An electronic device stores and transmits (internally and/or with otherelectronic devices over a network) code (which is composed of softwareinstructions and which is sometimes referred to as computer program codeor a computer program) and/or data using machine-readable media (alsocalled computer-readable media), such as computer or machine-readablestorage media (e.g., magnetic disks, optical disks, read only memory(ROM), flash memory devices, phase change memory) and computer ormachine-readable transmission media (also called a carrier) (e.g.,electrical, optical, radio, acoustical or other form of propagatedsignals—such as carrier waves, infrared signals). Thus, an electronicdevice (e.g., a computer) includes hardware and software, such as a setof one or more microprocessors coupled to one or more machine-readablestorage media to store code for execution on the set of microprocessorsand/or to store data. For instance, an electronic device may includenon-volatile memory containing the code since the non-volatile memorycan persist code/data even when the electronic device is turned off(when power is removed), and while the electronic device is turned onthat part of the code that is to be executed by the microprocessor(s) ofthat electronic device is typically copied from the slower non-volatilememory into volatile memory (e.g., dynamic random-access memory (DRAM),static random-access memory (SRAM)) of that electronic device. Typicalelectronic devices also include a set or one or more physical networkinterface(s) to establish network connections (to transmit and/orreceive code and/or data using propagating signals) with otherelectronic devices.

A movable object is an electronic device that includes one or morepropulsion units to propel the movement of the movable object. A movableobject can be an unmanned aircraft, an unmanned vehicle, or a robot. Onecommonality of these movable objects is that no humanpilot/driver/operator aboard to control these movable objects. That is,the movement of the movable object, using the one or more propulsionunits, is controlled through a different electronic device. An unmannedaircraft is also referred to as an unmanned aerial vehicle (UAV), adrone, or an unmanned aircraft system (UAS), all of which are usedinterchangeably referring to the unmanned aircraft herein.

An affiliated device is an electronic device that affiliates withanother electronic device in a video communication environment. In avideo communication environment, both the electronic device and theaffiliated device may be a wearable electronic device, a handheldelectronic device, or a movable object. The referred affiliation betweenthe affiliated device and the electronic device is typicallycommunicatively coupling (through a communication network) or connectingbetween the affiliated device and the electronic device (through one ormore wireline).

FIG. 1 is an exemplary illustration of a video communicationenvironment, in accordance with various embodiments of the presentdisclosure. As shown in FIG. 1, the video communication environment 100includes an electronic device 150, a communication network 190, and anaffiliated device 152.

The communication network 190 may be a variety of wireline or wirelessnetworks. The wireline between the electronic device 150 and theaffiliated device 152 includes one or more physical communication linkssuch as copper lines and optical fibers. The wireline network may deploytechnologies such as the universal asynchronous receiver/transmitter(UART) technology, the controller area network (CAN) technology, and theinter-integrated circuit (I2C) technology the wireless communicationnetwork may deploy technologies such as wireless local area network(WLAN) (e.g., WiFi™), Bluetooth, cellular the third/fourth/fifthgeneration (3G/4G/5G).

The electronic device 150 includes a video encoder 102. A camera 104 maybe implemented within or coupled to the electronic device 150. Thecamera 104 captures images and video, and the captured images and/orvideo are encoded by the video encoder 102. When a camera capturesvideo, the camera is sometimes referred to as camcorder or videorecorder, and in this specification, these terms are usedinterchangeably. While only one camera is illustrated, it is to beunderstood that the electronic device 150 may work with multiplecameras, the captured images and/or video from which are encodedaccording to one or more scheduling policy based on an operator request(e.g., the operator of the electronic device 150) and/or workloadcharacteristics (e.g., load balancing based on the workloads for thevideo encoder 102). In other words, the video encoder 102 may receivemultiple images and/or video feeds, and encode them so that they can betransmitted in more compact forms.

It is to be noted that encoding technologies for encoding video may beviewed as a superset that encompasses the encoding technologies forencoding images, as video being displayed is a stream of video frames,each being an image. Thus unless noted otherwise, the operationsdiscussed in this specification that are performed on video data applyto still image data too. Additionally, a camera may capture audio data,positional data along with the pictorial data, thus herein below thevideo data in this specification may include data of video frames, audiodata, positional data, and other information captured by one or morecameras.

The encoded data is then from the electronic device transmitted to theaffiliated device 152 through the communication network 190. Theaffiliated device 152 is another electronic device. At the affiliateddevice 152, the encoded data is decoded by a video decoder 112. Thedecoded data can then be viewed, e.g., on a display 114, which may bewithin the affiliated device 152 or coupled to the affiliated device152. When the encoded data includes audio data, the decoded data can belistened to from a speaker (not shown), singly or along with thedisplay. The video encoder 102 and video decoder 112 together are oftenreferred to as a codec system.

A codec system may support one or more video compression protocols. Forexample, the codec in the video communication environment 100 maysupport one or more of H.265 high efficiency video coding (HEVC), H.264advanced video coding (AVC), H.263, H.262, Apple ProRes, Windows MediaVideo (WMV), Microsoft (MS) Moving Picture Experts Group (MPEG)-4v3,VP6-VP9, Sorenson, RealVideo, Cinepak, and Indeo. While each videocompression protocol has its own advantages and drawbacks, the powerconsumption, coding/decoding speed, and/or picture fidelity are oftenthe most important factors to consider. Embodiments of the disclosureare not limited to a particular video compression protocol and areapplicable to video compression protocols that support slice encoding.

In one embodiment, the electronic device 150 is a mobile electronicdevice. For example, the electronic device 150 may be a wearableelectronic device, a handheld electronic device, or a movable object.When the electronic device 150 is a movable object, the camera 104 maybe an onboard camera, which takes aerial photographs and video forvarious purposes such as industrial/agricultural inspection, live eventbroadcasting, scientific research, etc.

A mobile electronic device generally has a smaller footprint incomparing to a stationary electronic device such as desktopcomputer/server, thus it is more challenging for a mobile electronicdevice to encode a large amount of video data in real-time. Yet viewer'sexpectation of video quality keeps going up. For example, nowadays it isadvantageous in the market place to provide 4K resolution video streamto a viewer. To provide the 4K resolution, which refers to a display ofhorizontal resolution on the order of 4,000 pixels and verticalresolution on the order of 2000 pixels, the camera 104 may be a 4Kcamera to capture the 4K resolution video. The 4K resolution for a videoframe typically includes 7˜11 M pixels, and each pixel may include 24bits, and the resulting video stream cannot be easily encoded in amobile electronic device, given its relatively small footprint allowedfor video encoding hardware.

Embodiments of the present disclosure provide solutions to address theseissues.

Embodiments of the present disclosure utilize intra-slice parallelencoding through one or more video encoding circuits. FIG. 2 is anexemplary electronic device including a video encoder according to oneembodiment of the disclosure. The electronic device 150 includes a videoencoder 102 and an external storage 210, and optionally camera 104.

The camera 104 may be a 4K camera, which provides video data in 4Kresolution, often referred to as 4K ultra-high-definition (UHD), whichhas a resolution such as 4096×2160 and 3840×2160 pixels. While 4Kresolution is used as an example of high definition (HD) resolution of avideo stream, embodiments of the disclosure are not limited to encodingof video streams in 4K resolution, and embodiments of the disclosure mayencode video data in other resolutions such as standard definition (SD)(e.g., 480 lines interlaced, 576 line interlaced), full high definition(FHD) (e.g., 1920×1080 pixels), 5K UHD (e.g., 5120×2880, 5120×3840,5120×2700 pixels), and 8K UHD (e.g., 7680×4320, 8192×5120, 10240×4320pixels). In other words, embodiments of the disclosure are notresolution specific, and they may be implemented to encode large amountof video data in real-time or near real-time.

The video encoder 102 includes a slice splitter 201, a frequencytransformer 202, a quantizer 204, a slice encoder 206, a rate controller205, a bit stream memory access unit (BSMAU) 208 (bit stream memoryaccess circuit), and a header memory access unit (HMAU) 209 (headermemory access circuit). In one embodiment, the video encoder 102 isimplemented as a semiconductor intellectual property (IP) block (alsoreferred to as an IP core) that is a reusable unit of logic, cell, orchip including integrated circuits (referred to simply as circuitshereinafter) in a die. An IP block is the intellectual property of oneparty, and that party may or may not be the same party that provides theelectronic device 150. It is to be noted that only entities mostrelevant to the embodiments of the disclosure are illustrated, and thevideo encoder 102 may include other modules/circuits. For example, avideo encoder often includes hardware such as image signal processor(ISP), microprocessors/cores, registers/caches/memory, and softwaremodules such as video process drivers to assist the circuits to processthe video streams.

While only one video encoder is illustrated, the electronic device 150may include multiple video encoders that encode video data from thecamera 104 or a different camera. The video encoder 102 receives videodata from the camera 104, and the encoding process of the video encoder102 is detailed herein.

Then the encoded data from the video encoder 102 is stored at anexternal storage 210, which is outside of the video encoder 102 thusreferred to as being “external”. The external storage may be one or moreof a variety of dynamic random-access memory (DRAM) such as double datarate synchronous DRAM (DDR SDRAM or referred to simply as DDR), singledata rate (SDR) SDRAM, static RAM (SRAM), persistent mass storage device(e.g., Flash, magnetic disk), and persistent memory such as Phase ChangeMemory (PCM), Phase Change Memory and Switch (PCMS), Memristor, andspin-transfer torque (STT) RAM.

The embodiments of the present disclosure aim at optimizing videoencoding without impacting the video decoding process. Thus, in oneembodiment, the embodiments of the present disclosure are implemented atthe electronic device 150 while the affiliated device 152 is unchanged.In other words, the affiliated device 152 is unaware of the improvementin the electronic device 150 as the encoded data received at theaffiliated device 152 is the same, with or without embodiments of thepresent disclosure implemented in the electronic device 150.

Hiding the changes of the encoding process from the decoding at theaffiliated device 152 is advantageous. The decoding process is oftenspecified in a standard, which defines how encoded data stream is to bedecoded. With the embodiments of the present disclosure changes only howthe encoding is performed without impacting the decoding process, thevideo decoder 112 remains the same thus the embodiments of the presentdisclosure are compatible with existing video decoder and makes theadoption of the embodiments easier.

FIG. 3 illustrates video encoding processes according to one embodimentof the disclosure. FIG. 3 is similar to FIG. 2, and the same referencesindicate elements or components having the same or similarfunctionalities. Certain aspects of FIG. 2 have been omitted from FIG. 3in order to avoid obscuring other aspects of FIG. 3.

Each of the frequency transformer 202, quantizer 204, slice encoder 206,and rate controller 205 may be implemented in circuits, and togetherthey are referred to as video encoding circuits 350. Note the videoencoding circuits 350 may include other circuits and components, andsome of the illustrated circuits within the video encoding circuits 350may be integrated with each other or with other circuits thus arereferred to using different terms.

Task box 1-5 illustrate the order in which operations are performedaccording to one embodiment of the disclosure.

At task box 1, the slice splitter 201 splits video data into a pluralityof slices. The video data is received from the camera 104. Each slicecontains a plurality of data blocks, each data blocks containing aplurality of data points, which are processed in a number of datastreams. The slice splitter 201 may be implemented as asoftware/firmware module in one embodiment. In an alternativeembodiment, the slice splitter 201 may be implemented using one or morecircuits.

The slice splitting process may be illustrated in FIG. 4A, whichillustrates data blocks being processed through encoding according toone embodiment of the disclosure. Video data includes a sequence ofvideo frames that are received from a camera such as the camera 104 at acertain rate. For example, the video data may be a video stream of 15,30, 60, or 120 frames per second received from the camera 104.

FIG. 4A illustrates a video frame 402, which includes a luma (Y′)channel 410, and two chroma channels of the blue chroma (Cb) and redchroma (Cr) channels 412 and 414. Y′ is distinguished from Y, which isluminance that encodes light intensity nonlinearly based on gammacorrected red green blue (RGB) primaries. Y′CbCr color space is definedby a mathematical coordinate transformation from an associated RGB colorspace.

The scopes of the terms Y′CbCr, YCbCr, Y′UV, YUV, and YPbPr aresometimes ambiguous and overlapping, and the embodiments of the presentdisclosure are not limited to a particular color space scheme. Forexample, the illustrated Cb and Cr channels in FIG. 4A may be replacedwith U and V channels. The description of the disclosure uses Y′CbCr asan example, and it will be apparent to those skilled in the art thatother color spaces may utilize the embodiments of the disclosure.

Within the video frame 402, macroblocks are defined for data processing.Each macroblock includes data blocks from the luma channel and the twochroma channels. The illustrated sampling ratio is 4:4:4, where eachchannel has the same amount of data blocks in a macroblock. Asillustrated, with the sampling ratio of 4:4:4, the macroblock 428includes 4 data blocks from each of the luma channel and the two chromachannels. The data blocks within different channels may be ordereddifferently. In this example, the data blocks in the luma channel areordered horizontally first and then vertically, while the data blocks inthe two chroma channels are ordered vertically first and thenhorizontally.

While the sampling ratio of 4:4:4 is illustrated, other sampling ratiossuch as 4:2:2, 4:2:0, 4:1:1 are also widely used. With differentsampling ratios, different numbers of data blocks from the luma andchroma channels may form a macroblock. For example, a macroblock with asampling ratio of 4:2:0 may include 4 data blocks from the luma channeland 1 data block each from each chroma channel, while a macroblock witha sampling ratio of 4:2:2 may include 4 data blocks from the lumachannel and 2 data blocks each from each chroma channel.

Note that sometimes the sampling ratios are denoted without semicolonbetween the numbers such as 444, 420, and 411. Additionally, some videocodec includes an alpha channel, which is an additional image channel(e.g., extending an RGB image) or standalone channel controlling alphablending that combining a translucent foreground color with a backgroundcolor, thereby producing a new blended color). When the alpha channel isincluded, the sampling ratio may be denoted with four numbers, such as4:4:4:4 or 4444 for the macroblocks with equal amount of data blocks ineach channel.

A data block often includes 8×8=64 data points. Each data point may be apixel. Each pixel may be represented by 1˜48 bits. 24 bits per pixelsometimes is referred to as true color, 30/36/48 bits per pixelsometimes is referred to as deep color, and they are commonly deployed.As illustrated, a data block in FIG. 4A includes 4×4=16 data points, andeach data point has a value of Ymn, where m is the row number and n isthe column number of the data block within the video frame 402. Thus,Y03 is the luma data value (e.g., represented by one of 1˜48 bits) atcolumn 0, row 3 as illustrated at reference 452. The chroma data valuesmay be represented similarly as luma data values, and they are notillustrated in FIG. 4A due to space limitation.

The embodiments of the disclosure are not limited to a particularsampling ratio or a particular form of a data block. While FIG. 4Aillustrates the sampling ratio of 4:4:4 and each data block having4×4=16 data points, the example is for simplicity of illustration, andit will be apparent to those skilled in the art that other samplingratios and other data block formation may utilize the embodiments of thedisclosure.

The video frame 402 is a key frame (also referred to as an intra-frameor I-frame) that contains data points of a complete image. A videostream may also include P-frames (predicted picture frames) that includechanges from a previous frame and B-frames (bi-predictive pictureframes) that includes change from the previous frame and followingframe. Each P-frame and B-frame may be represented by a macroblock and amotion vector. The details of the compression of P-frames and B-framesare known in the art and not repeated here. It is to be noted thatembodiments of the disclosure are not limited to a particular types offrames, and a video stream may contain a variety of I-frames, P-frames,B-frames, or predictive frames.

A slice contains one or more contiguous rows of macroblocks. Forexample, in FIG. 4A, the macroblocks are ordered in rows. A slice startsat the first macroblock in a row and ends at the last macroblocks in thesame or another raw. Thus a slice contains an integer number of completerows. A slice is byte-aligned, and slices may be transmittedconcurrently. Slice 420 includes macroblocks of rows 1-3, slice 422includes macroblocks of rows 4-6, and slice 424 includes macroblocks ofrows 7-8. As illustrated, the slices may not have the same amount ofmacroblocks. Within a macroblock, the blocks in the luma channel andchroma channels may be numbered in a raster scan order or a zig-zag scanorder. For example, the data blocks in macroblock 428 luma channel arenumbered in a raster scan order, where the data blocks are numbered onedata point line at a time, and the data blocks on the second data pointline are numbered following the end of the number of the first datapoint line. In contrast, the data blocks in macroblock 428 chromachannels are numbered in a zig-zag scan order, which starts at the firstdata block being Block 0, Block 1 being at the column 0, row 1, Block 2being at the column 1, row 1, and Block 3 being at the column 1, row 1.That is, the blocks are numbered gradually further away from the block0. The sequence order of the data blocks (illustrated using sequencenumbers of the data blocks) is used for scanning as discussed hereinbelow. It is to be noted while 3 slices are illustrated in the figure, avideo frame may contain many more slices. For example, a video frame maybe divided into 32 slices.

A slice is a portion of a video frame, and encoding a video frame onslice based enables a codec to achieve the right trade-off between errorresilience and compression. The data blocks within a slice may beintra-coded (referred to as intra-slice encoding), and the slice may bereconstructed at the decoder regardless of errors in other slices of thevideo frame. Thus while slice-based encoding generates additionalencoding information at the slice level that needs to be decoded, suchcodec system provides better error resilience.

With slice splitting, inter-slice encoding may be implemented so thatmultiple slices may be encoded concurrently through video encodingcircuits. Embodiments of the disclosure go steps further, and describehow intra-slice encoding may be performed so that a single slice may beencoded concurrently using multiple data streams.

Referring back to FIG. 3, once the video data is split into theplurality of slices, the slices are provided to the frequencytransformer 202, where the values of the data blocks in the slices aretransformed into frequency coefficients (also referred to as spectraldata coefficients) at task box 2. The frequency transformer 202 convertsthe values of the data blocks, which are spatial domain videoinformation (luma data and chroma data), into frequency domain data, thefrequency coefficients. The frequency transformer 202 may applyfrequency transformation on a data block with 8×8, 8×4, 4×8, 4×4, orother size of data points.

The frequency transformer 202 may be a discrete cosine transform (DCT)transformer in one embodiment. Through a DCT operation, data points of adata block are transformed to a set of frequency coefficients of equalsize. That is, a DCT operation on a 4×4 or 8×8 data block yieldsfrequency coefficients of 4×4 or 8×8 block respectively. For most videoframes, the image information lies at low frequencies which appear inthe upper-left corner of the DCT-encoded block. The lower-right valuesof the DCT-encoded block represent higher frequencies, and may be smallenough to be ignored with little visible distortion. The top left cornervalue in the DCT-encoded block is the DC (direct current, i.e.,zero-frequency) component and lower and right more entries representlarger vertical and horizontal spatial frequencies.

The DCT operation is a separable transformation in that a matrix for theDCT transformation may be decomposed into two matrices, one thatcorresponds to a column transformation and another that corresponds to arow transformation. Thus, the DCT operation may be implemented as twoone-dimensional (1D) transformations. Thus, a two-dimensional (2D) DCTis just a 1D DCT applied twice, once in the column direction and once inthe row direction. In other words, the frequency transformer 202 mayperform a first 1D DCT operation in the column direction of the provideddata block, and then perform a second 1D DCT operation in the rowdirection of the 1D-encoded block to produce a 2D-encode block,containing the frequency coefficients. Alternatively, the first 1D DCToperation may be performed in the row direction of the block and thesecond 1D DCT operation performed in the column direction of the block.

In one embodiment, the frequency transformer 202 performing DCT isimplemented in one or more circuits. For an 8×8 data block, thefrequency transformer 202 takes 8 data points concurrently in one clockcycle. Since the operation on the data points at each 1D DCTtransformation takes one clock cycle, the 2D transformation decomposedinto 2 1D DCT transformation takes two clock cycles for the 8 datapoints. Thus, the DCT operations on the 8×8 data block takes 16 clockcycles. Similarly, for a 4×4 data block, when the frequency transformer202 takes 4 data points concurrently in one clock cycle, the DCToperation on the 4×4 takes 8 clock cycles.

In one embodiment, the frequency coefficients produced from thefrequency transformer 202 are scanned first, and then the scannedfrequency coefficients are quantized by the quantizer 204. In analternative embodiment, the frequency coefficients produced from thefrequency transformer 202 are quantized first, and then scanned. Eithersequence may be implemented in embodiments of the disclosure.

In one embodiment, the scan is performed for all the frequencycoefficients of a slice. The scan may be performed in a frequency-firstorder. FIG. 4A illustrates scanning of the frequency-first orderaccording to one embodiment of the disclosure.

Referring to FIG. 4A, the data points in different data blocks aretransformed through the frequency transformation at reference 492. Thenthe resulting frequency coefficients from the frequency transformationis scanned at slice scanning 494, which is done in a frequency-firstorder. In one embodiment, in the frequency-first order, the frequencycoefficients are ordered with frequency coefficients at the sameposition of each data block of a slice being grouped together, and forthe frequency coefficient at the same position, the one of the blockwith lower block sequence order is scanned earlier.

As illustrated at reference 494, the frequency coefficients of a wholeslice in one channel (say slice 1 luma channel) are ordered in a matrix.The matrix has a width equals to the number of data blocks in the slice,and depth equals to the number of data points in a data block. Thefrequency coefficients are ordered based on the scanning method. Forexample, the frequency coefficients of the DC component (the lowestfrequency) in all the data blocks in the slice are ordered in the raw 0,and the frequency coefficient for Block 0 DC component is the first oneto be scanned, followed by the DC component for Block 1, and the row 0ends at the DC component for Block N-1, where N is the number of totaldata blocks in the slice. The end of row 0 is followed by row 1 in thefrequency-first scanning order, where row 1 contains the frequencycoefficients for next lowest frequencies (higher than the DC componentbut lower than all other frequencies) of all the data blocks. The row 1is ordered starting with that next lowest frequency coefficient forBlock 0, followed by the one for Block 1, and so on.

Thus, the frequency coefficients are ordered to be scanned with the onespositioned corresponding to data point 0 (the DC components of thefrequency coefficients in the frequency domain), followed by the onespositioned corresponding to the next lowest frequency, until ends at thehighest frequency at M-1, where M is the total number of data points ina data block (the frequency coefficients are ordered to correspond tothe increasingly higher frequencies). Since the scan order starts withthe DC component of all data blocks and following the data point orderto higher frequencies, the scan order is referred to as afrequency-first scanning. It is to be noted that since thefrequency-first scanning groups scans the frequency coefficients at afirst same frequency of all data blocks in a slice first, and them moveto the frequency coefficients at a second, higher frequency of all datablocks in the slice, the frequency coefficients of all the blocks in theslice are scanned prior to the operations at the next step such asquantization.

In alternative, a slice may be scanned in a block-first order. FIG. 4Billustrates data blocks being processed through encoding according toanother embodiment of the disclosure. FIG. 4B is similar to FIG. 4A, andthe same references indicate elements or components having the same orsimilar functionalities.

FIG. 4B illustrates a block-first order scanning according to oneembodiment of the disclosure. As discussed earlier, at reference 492,the data points in different data blocks are transformed through thefrequency transformation. Then the resulting frequency coefficients fromthe frequency transformation is scanned at slice scanning 484, which isdone in a block-first order. In one embodiment, in the block-firstorder, the frequency coefficients are ordered with frequencycoefficients of one data block of a slice are grouped together, wherethe frequency coefficients of the data block are scanned from lowerfrequencies to higher ones; and between data blocks, the frequencycoefficient are scanned in the order of data block sequence order (thedata block sequence order is discussed herein above in relation to themacroblock 428).

At reference 484, the frequency coefficients of Block 0 of a slice arein the highest scanning order in the block-first slice scanning, and thelast block of the slice, Block N-1 are in the lowest scanning order inthe block-first slice scanning. Within a block, the lower frequenciesare scanned first, followed by the higher frequencies. For example,within Block 0, the DC component (the lowest frequency) is scannedfirst, followed by the second lowest frequency, whose position maycorrespond to Data point 1 of Block 0, followed by the third lowestfrequency, whose position may correspond to Data point 2 of Block 0, andso on. That is, all the frequency coefficients within Block 0 arescanned first, followed by the frequency coefficients within Block 1,until all the frequency coefficients within the slice are scanned. Inthe block-first slice scanning, after each data block finishes DCToperation and generates the frequency coefficients, the frequencycoefficients of that data block, without waiting for data fromprocessing the subsequent data blocks, may be provided to the operationsat the next step such as quantization. Thus, while in thefrequency-first slice scanning all the frequency coefficients in a sliceare scanned prior to the next operation on any of the frequencycoefficients, in the block-first slice scanning the frequencycoefficients in one block may be scanned and then provided to the nextstep.

After the frequency-first or block-first slice scanning, the frequencycoefficients may be provided to quantization. Referring back to FIG. 3,at task box 3, the quantizer 204 quantizes the frequency coefficients ofa slice, using either uniform, scalar quantization with a step-size thatmay vary on a frame-by-frame basis or other basis. Alternatively, thequantizer 204 may apply another type of quantization to the frequencycoefficients, e.g., a non-uniform, vector, or non-adaptive quantization.Quantization is a lossy operation, and information lost duringquantization cannot be recovered at the decoder.

The quantization may happen prior to or after the slice scanning,depending on implementation. In one embodiment, the quantization rate ofquantizer 204 matches the transforming rate of the frequency transformer202. For example, when the frequency transformer 202 processes 8 datapoints every two clock cycles as indicated in a previous example, wherethe frequency transforming rate is equivalent to 4 data points per clockcycle, the quantizer 204 may quantize 4 data points per clock cycle,thus the processing rate between the frequency transformer 202 andquantizer 204 are matched.

Then at task box 4, the slice encoder 206 encodes the quantizedfrequency coefficients in a number of data streams concurrently. Theslice encoder 206 compresses the output of the quantizer 204 such asfrequency coefficients as well as coding information such asquantization step size and motion information. In one embodiment, theslice encoder 206 uses entropy encoding, which may combine a number ofconsecutive zero valued quantized coefficients with the value of thenext nonzero quantized coefficient into a single symbol, and also hasspecial ways of indicating when all of the remaining quantizedcoefficient values are equal to zero. The entropy coding method may usevariable length coding tables. The entropy encoding may use at least oneof Huffman coding, arithmetic coding, or static coding such as universalcoding or Golomb coding (e.g., exponential-Golomb coding or justExp-Golomb coding and Rice coding).

In one embodiment, the encoding rate of the slice encoder 206 matchesthe quantization rate of quantizer 204. For example, when the quantizer204 quantizes 4 data points per clock cycle, the slice encoder 206encodes the 4 data points per clock cycle. The result is that 4concurrent data streams are encoded. FIG. 4C illustrates concurrentencoding matching quantization according to one embodiment of thedisclosure. The data streams 352-358 arrive at the quantizer 204 (e.g.,directly after the frequency transformer 202 or after scanning asdiscussed herein above), and the frequency coefficients are quantized atquantization units 1-4 at references 432-438 in one embodiment. Theconcurrently quantized frequency coefficients are then provided to theslice encoder 206 (through a post quantization scanning in someembodiments), which contains encoder units 1-4 at references 472-478.Once the encoding is completed, the data in the number of data streamsare multiplexed together through a slice data multiplexor 302. While theslice data multiplexor 302 is illustrated within the slice encoder 206,the slice data multiplexor may be implemented outside of the sliceencoder 206. For example, the slice data multiplexor 302 may beimplemented in a circuit that multiplexes the concurrently encoded datastreams after the data streams exits the slice encoder 206 (e.g., themultiplexor may be implemented within the BSMAU 208) in one embodiment.

One feature of the slice encoder 206 is that it encodes incoming data ofa slice concurrently (thus it may also be referred to as a “parallel”slice encoder). That is, the concurrent encoding is performed within aslice, and the intra-slice concurrent encoding provides better encodingefficiency. In one embodiment, the number of concurrent data streams atthe slice encoder 206 matches the concurrency of the transforming rateof the frequency transformer 202 and/or quantization rate of thequantizer 204. Thus, when the frequency transformer 202 processes 8 datapoints every two clock cycles as indicated in the previous example,where the frequency transforming rate is equivalent to 4 data point perclock cycle, the number of concurrent data streams at the slice encoder206 is 4.

FIG. 4A illustrate the intra-slice concurrent encoding according to oneembodiment of the disclosure. As discussed herein above, the frequencycoefficients within a slice are scanned frequency-first at reference494, and the scanning may happen prior to or after the quantization. Thequantized frequency coefficients are then encoded concurrently.Following the previous example, the number of the concurrent datastreams is 4. Similarly, when the frequency transformer 202 processes 4data points per clock cycle (same as the rate at 8 data points per twoclock cycles), the number of the concurrent data steams is still 4. Theslice encoder 206 performs concurrent data stream encoding for a sliceat reference 496. Each of the concurrent data streams is encoded in apath, thus for slice 1, the encoding is performed through paths 1-4 atreference 442-448 concurrently.

At each clock cycle, the slice encoder 206 takes 4 quantized frequencycoefficients sequentially from the results of the frequency-first slicescanning of slice 1 at reference 496. As illustrated, Cx,y indicates afrequency coefficient corresponds to data block x at frequency point y.Thus, the first 4 frequency coefficients to be encoded according to thefrequency-first scanning order are C0,0 C1,0, C2,0, and C3,0, which areall the DC components (zero frequency) of frequency coefficientscorresponding to Blocks 0, 1, 2, and 3 of the slice. The first 4frequency coefficients are encoded concurrently at the paths 1-4,followed by the next 4 frequency coefficients, which are the next 4 DCcomponents of frequency coefficients corresponding to Blocks 4-7 of theslice. After the DC components of all blocks are encoded, the sliceencoder 206 encodes the next lowest frequency, the first 4 of which areC0,1, C1,1, C2,1, and C3, 1. The process continues until all thefrequency coefficients of the slice are encoded concurrently.

While FIG. 4A illustrates the intra-slice concurrent encoding for thefirst-first slice scanning, FIG. 4B illustrates another intra-sliceconcurrent encoding process. As discussed herein above, the frequencycoefficients within a slice are scanned block-first at reference 484,and the scanning may happen prior to or after the quantization. Thequantized frequency coefficients are then encoded concurrently.Following the previous example, the number of the concurrent datastreams is 4.

Referring to FIG. 4B, at each clock cycle, the slice encoder 206 takes 4frequency coefficients sequentially from the results of the block-firstslice scanning of slice 1 at reference 486. Thus, the first 4 frequencycoefficients to be encoded according to the block-first scanning orderare C0,0 C0,1, C0,2, and C0,3, which are the first 4 frequencycoefficients of Block 0. The first 4 frequency coefficients are encodedconcurrently at the paths 1-4 (illustrated at references 462-468),followed by the next 4 frequency coefficients, which are the next 4frequency coefficients of Block 0. After all the frequency coefficientsof Block 0 are encoded, the slice encoder 206 encodes Block 1, which hasthe first 4 frequency coefficients being C1,0, C1,1, C1,2, and C1,3. Theprocess continues until all the frequency coefficients of the slice areencoded concurrently.

Once the data in a slice is encoded through the encoding of theconcurrent data streams, it is then combined into a single data stream.The combination may be performed by a multiplexor. In one embodiment,the multiplexor is within the slice encoder 206 such as the a slice datamultiplexor 302. The encoding of the concurrent data streams may resultin different lengths of encoded data. For example, for slice 1, thelengths of encoded data in paths 1-4 may be different from each other,and a variety of approaches may be implemented to align the encoded dataso that they can be combined properly as a single data stream. Forexample, bit shifting and inserting dummy bits may be implemented in oneor more of the paths.

The rate of the encoding is coordinated with the rate of quantization inone embodiment of the disclosure. Referring back to FIG. 3, the ratecontroller 205 is configured to control the quantization based on atleast the information from the slice encoder 206. The information fromthe slice encoder 206 may include the length of the encoded data for oneslice and/or the clock cycles taken to encode of the slice.Additionally, information from the ISP and registers of the videoencoding circuits 350 may also control the quantization at quantizer204. The rate controller 205 may control the step-size (often beingreferred to using the symbol Q) of quantization. In one embodiment, thecontrol of the step-size Q is determined through a quantizationparameter (QP) when the codec complies with the H.264 protocol. In oneembodiment, the rate control uses the information from encoding of thecurrent slice or several slices that have been encoded to control thestep-size of the subsequent slices.

Embodiments of the disclosure provide encoding efficiency with matchingconcurrency at steps of the encoding process. The encoding operatingparameters 320 illustrate one example of such matching concurrency ofthe video encoding circuits 350. As illustrated, the video encodingcircuits 350 may operate at frequency 400 MHz (clock of 2.5 nanosecondsper cycle); the frequency transformer 202 operates on 8 data pointsevery two clock cycles; the quantizer 204 operates on 4 data pointsevery clock cycle; and the slice encoder operates on 4 data points everyclock cycles through the concurrent data streams).

The encoding operating parameters 320 include a common operating clockfor all the video encoding circuits. However, the matching concurrencyof the video encoding circuits does not depend on all the video encodingcircuits operating on the same operating clock. For example, thefrequency transformer 202 may operate at twice of the quantizer 204,thus instead of 8 data points every two clock cycles of the 400 MHzclock, it operates at 800 MHz clock and processes 8 data points everyone clock cycle of the 400 MHz clock on which the quantizer 204operates. In that case, the quantizer 204 needs to realign to operate on8 data points every clock cycle of 400 MHz clock, and the slice encodermay split the data streams into 8 paths, each for the 8 data pointstaken at a clock cycle of the 400 MHz.

The matching concurrency of operating depends on the frequencies ofoperating clock at different circuits and the clock cycles taken toperform the operations in the video encoding circuits. In theillustrated encoding operating parameters 320, because the frequencytransformer takes two clock cycles to process a group of data points, ittakes twice amount of data points from the data stream at one time incomparison to the later stages. The data point consumption differs whenthe frequency transformer process cycle is different.

FIG. 5 is a flow diagram illustrating the intra-slice encoding accordingto one embodiment of the disclosure. The method 500 may be implementedin an electronic device such as the electronic device 150.

At reference 502, the electronic device optionally obtains video datafrom a camera unit such as the camera 104. The camera unit, within orcoupled to the electronic device, generates video data. In oneembodiment, the camera unit contains video/image compression componentsthat perform an initial video/image compression on the generated videodata. For example, the compression components of the camera unit mayremove inter-frame redundancy through encoding.

At reference 504, the electronic device splits the video data into aplurality of slices. Each slice contains a plurality of data blocks, andeach data block contains a plurality of data points, which are processedin a plurality of data streams. In one embodiment, the plurality of datablocks forms macroblocks, where one macroblock includes one or more datablocks from a luminance (Y) channel and two chrominance (Cr and Cb)channels. In one embodiment, the operations at reference 504 areperformed by a slice splitter, which may be implemented as one or moreof circuits or software/firmware modules.

At reference 506, the electronic device encodes the data blocks in theplurality of data streams concurrently and combines the encoded datastreams into a combined data stream. The encoding is performed using oneor more video encoding circuits. In one embodiment, the combination ofthe encoded data streams is performed outside of the one or more videoencoding circuits.

Then at reference 508, optionally the electronic device stores thecombined data stream in an external storage such as the external storage210 discussed herein above. In one embodiment, the operations withinreferences 504 and 506 are performed by a video encoder.

After the operations of method 500, the combined data stream is thentransmitted from the external storage to a decoder such as the videodecoder 112, where the combined data stream may be decoded. The decodeddata stream is then available for viewing or further process at anaffiliated device such as the affiliated device 152.

In one embodiment, the video data from the camera unit may be providedto a plurality of video encoders, and each a portion of the video data.For example, when the camera unit provides video data for 8K UHDresolution at 30 frame per second (fps), one video encoder may encodeone half of the video data (e.g., one video encoder encodes the leftportions the video frames while the other encodes the right portions).Such encoding results in two combined data streams. The two combineddata streams are then integrated in the external storage outside of thetwo video encoders, and the resulting data stream is transmitted to thedecoder.

FIG. 6 is a flow diagram illustrating encoding operations in moredetails according to one embodiment of the disclosure. The method 600may be implemented in an electronic device such as the electronic device150. In one embodiment, the method 600 is an implementation of reference506.

At reference 610, the electronic device performs frequencytransformation on the data blocks transforming the data blocks into aplurality of frequency coefficients. In one embodiment, the frequencytransformation is performed through DCT.

At reference 612, the electronic device quantizes the plurality offrequency coefficients. As discussed herein above, the plurality offrequency coefficients transformed from the plurality of data blockswithin a slice may be scanned in a frequency-first order or ablock-first order, and the scanning may be performed prior to or afterthe quantization.

At reference 614, the electronic device encodes the plurality offrequency coefficients in a plurality of data streams concurrently. Inone embodiment, the combined encoding rate to concurrently encode theplurality of data streams matches a combined rate of the frequencytransform of the data blocks as discussed herein above.

At reference 616, the electronic device optionally controls thequantization based on at least information from the encoding. In oneembodiment, the information from the encoding include the length of theencoded data for one slice and/or the clock cycles taken to encode ofthe slice. Additionally, information from an ISP and registers of thevideo encoding circuits performing the encoding may also control thequantization.

It is to be noted while in one embodiment, the data blocks are frequencytransformed and quantized prior to being encoded, in an alternativeembodiment, the data blocks may be encoded directly without one or moreof the frequency transformation and quantization.

Embodiments of the disclosure encode a plurality of data blocks within aslice concurrently in a plurality of data streams. The encoded data maybe stored within the video encoder that performs the encoding firstprior to being transmitted to an external storage. FIG. 7 illustrates apost-encoding process according to one embodiment of the disclosure.While in one embodiment, the operations in FIG. 7 are a continuation ofoperations in FIG. 3, where the video data are encoded through thespecified process, in an alternative embodiment, the operations in FIG.7 are independent from the operations in FIG. 3 and they are to processencoded data through a different intra-slice encoding process, using thesame or different elements or components as illustrated in FIG. 3.

FIG. 7 contains elements or components with the same reference numbers,which indicate the elements or components having the same or similarfunctionalities. Task boxes 1-5 illustrate the order in which operationsare performed according to one embodiment of the disclosure. FIG. 7illustrates a portion of an electronic device performing video encoding.

Referring to FIG. 7, the BSMAU 208 includes a slice header formationmodule 704 (slice header formation sub-circuit), video data buffer 706,and optional slice data multiplexor 702. The slice data multiplexor 702multiplexes concurrently encoded multiple data streams of slices, if themultiplexing is not done in the earlier stage of the video encoder.

At task box 1, the electronic device stores the encoded data of a slicein the video data buffer 706. The video data buffer is a single port RAMin one embodiment. While receiving the encoded data in the BSMAU 208, attask box 2, the slice header formation module identifies the sliceheader information of the slice. The slice header information may beobtained directly from the encoding process and/or from the process ofmultiplexing the concurrently encoded multiple data streams of a slice.

In one embodiment, the slice header information of the slice includes atleast one of a slice header size, a size of luma data in the slice, or asize of chroma data in the slice. Additionally or alternatively, theslice header information of the slice includes one or more quantizationparameters such as a scale factor of the slice for quantization in oneembodiment. FIG. 8 illustrates composition of slice header informationaccording to one embodiment of the disclosure. As illustrated, the sliceheader information parameters 802 include values 804, which are thefollowing: the slice header size in bytes 810 has a value of 2indicating 2 bytes, the scale factor 812 has a value of 4, the size ofluma data in the slice in bytes 814 has a value of 186 indicating 186bytes, and the size of chroma data in the slice in bytes 816 has a valueof 62 indicating 62 bytes. The slice header information may have more,less, or different parameters as illustrated in FIG. 8 in variousembodiments of the disclosure.

At task box 3, the electronic device stores the slice header informationeither in the video data buffer 706 together with the encoded data ofthe slice or directly transmit it to the external storage 210. Thedecision of either to store the slice header information in the bufferor directly transmit to the external storage is based on at leastpartially on the combined length of the slice header information and theencoded data of the slice. If the storage space within the buffer isinsufficient to accommodate both the slice header information and theencoded data of the slice, the slice header information is transmittedto the external storage directly, otherwise the slice header informationis stored along with the encoded data of the slice.

Storing the slice header information along with the encoded data of theslice is advantageous in that the slice header information then will betransmitted along with the encoded data of the slice (the slice headerinformation may be transmitted ahead of the encoded data of the slice),and they can be written to the external storage in one or moreconsecutive clock cycles, thus speed up the processing of the slice.

Once the encoded data of the slice is stored in the buffer, its sliceinformation is known. Thus, at task box 4, the slice information of theslice is stored in a memory location at HMAU 209, which is related tobut different from the video data buffer. The slice information includesthe slice length after encoding, thus the slice information is notavailable until the slice encoding completes. It is advantageous tostore the slice information in a memory location different from the onestoring the encoded data so that the encoded data of a slice can becontinuously written to the buffer without concerning about the yet tobe determined slice length. Once the slice length is determined, it iswritten to the HMAU 209. The slice information is written through aslice information formation module 712. The slice information formationmodule 712 writes slice information to a slice information buffer 714,which may be implemented using a single port RAM (SPRAM). In oneembodiment, the slice information includes a slice index table.

FIG. 9 illustrates a slice index table according to one embodiment ofthe disclosure. Within the slice index table, each slice is indexed witha numeric number, and the slice index table indicates the slice lengthsof all the slices in a frame such as the video frame 402. The sliceindex table includes slice index 902 and slice length 904. Each entry inthe slice index table indicates a slice length of a slice. In oneembodiment, the slice length is indicated using two bytes. In thisexample, slice 1 at 912 has the slice length of 486 bytes, slice 2 at914 has the slice length of 359 bytes, slice 3 at 916 has the slicelength of 378 bytes, and slice 4 at 918 has the slice length of 186bytes.

The slice index table is filled in as slices within a frame are encoded.The value of each entry for a slice is filled in once the correspondingslice completes its encoding. Thus, the slice index table is notcompleted until all the slices within the frame are encoded. In oneembodiment, each entry of the slice index table includes additionalinformation. For example, the entry may include which video frame thatthe slice is sourced from. It is noted that slice information and sliceheader information are distinctive from each other in that sliceinformation is the information of the slice in relationship with a frameto which the slice belongs (e.g., slice information stored in the sliceindex table to provide a summary of lengths of slices within a frame asillustrated in FIG. 9), while the slice header information is theinformation regarding intra-slice encoding (e.g., informationillustrated in FIG. 8).

Referring back to FIG. 7, at task box 5, either the B SMAU 208 or theHMAU 209 transmits its data to the external storage 210 through the oneor more bus connections upon obtaining a permission. The one or more busconnections at reference 750 may be access through arbitration, thewinner of which obtains the permission. The one or more bus connectionsmay be implemented as parallel bus lines that concurrently transmitmultiple bits (e.g., 128 bits) from either the BSMAU 208 or the HMAU209. For example, the bus lines may be an advanced extensible interface(AXI) in the ARM architecture. The parallel bus lines transmission issometimes referred to as burst, and during a burst, data may betransmitted to the external storage based on a single address.

The BSMAU 208 or the HMAU 209 may issue a request when it is ready towrite to the external storage 210. An arbitrator of the bus connectionsdetermines if the bus connections are idle, and if the bus connectionsare idle, the BSMAU 208 or the HMAU 209 takes the ownership of the busconnections (e.g., through setting up a lock), and transmits its data tothe external storage 210. Once the transmission completes, the MAUreleases the bus connections, and others may use the bus connections.

In one embodiment, the BSMAU 208 requests and then transmits the storeddata in the buffer to the external storage when a predeterminedcondition is met. The predetermined condition may be that the storeddata in the buffer has reached a threshold (e.g., 85% full) and/or theslice header information of a slice is stored in the buffer.

FIG. 10 is a flow diagram illustrating transmitting data to an externalstorage according to one embodiment of the disclosure. Method 1000 maybe implemented in an electronic device such as the electronic device150.

Method 1000 starts at reference 1002, where the electronic device storesdata from encoding in a buffer at the first memory location. The buffermay be the video data buffer 706 as discussed herein above. The datafrom encoding may include both the data encoded from the combined datastream and slice header information of a slice as discussed herein.

At reference 1004, the electronic device may store slice information(such as a slice index table) from the encoding in a second memorylocation relative to the first memory location. The two memory locationsare different but related in that both store information related tointra-slice encoding of slices of a video stream, and the electronicdevice may identify the two memory locations as related.

At reference 1006, it is determined a permission to transmit data viathe one or more bus connections to an external storage such as externalstorage 210. Either the BSMAU 208 or the HMAU 209 may request thepermission to transmit, and the request may be triggered after apredetermined condition is met at the buffer or the second memorylocation. For example, for the BSMAU 208, the predetermined conditionmay be that the stored data in the buffer has reached a threshold (e.g.,85% full) and/or the slice header information of a slice is stored inthe buffer; for the HMAU 209, the predetermined condition may be thatthe slice index table for a frame is complete or the stored data in thesecond memory location has reached a threshold. The determination may bemade by an arbitration module monitoring access request of the one ormore bus connections.

When it is determined that the buffer obtains the permission to transmitits data, the flow goes to reference 1008, where the stored data fromthe buffer is transmitted to the external storage. When it is determinedthat the second memory location obtains the permission to transmit data,the flow goes to reference 1010, where the slice information istransmitted to the external storage.

FIG. 11 is a flow diagram illustrating storing data in a video encodingbuffer according to one embodiment of the disclosure. Method 1100 may beimplemented in an electronic device such as the electronic device 150.The video encoding buffer may be the video encoding buffer 706 asdiscussed herein above.

At reference 1102, the electronic device stores data in a combined datastream of a slice in a buffer at a first memory location. The combineddata stream is a data stream being combined from a plurality of datastreams concurrently encoded by a video encoder. The combination may beperformed within the video encoder (such as the slice data multiplexor302), or by a multiplexor at the first memory location such as the slicedata multiplexor 702 of the BSMAU 208.

At reference 1104, the electronic device identifies slice headerinformation. The slice header information may be obtained during theencoding of the plurality of data streams (e.g., from the slice encoder206) and/or during combining the data streams (e.g., from the slice datamultiplexor 302 or 702).

At reference 1106, it is determined whether the buffer is sufficient toaccommodate a slice's slice header information and the data of the slicein the combined data stream. The determination may be based on thestorage capacity of the buffer. When the slice header information andthe data of the slice in combination are within the storage capacity (ora certain threshold of the storage capacity) of the buffer, the flowgoes to reference 1108, where the slice header information of the sliceis stored in the buffer along with the data of the slice. Afterward,when a predetermined condition is met, the data in the buffer istransmitted to the external storage as discussed herein above.

When the slice header information and the data of the slice incombination exceeds the storage capacity (or a certain threshold of thestorage capacity) of the buffer thus the buffer is insufficient toaccommodate both of the slice header information and the data of theslice, the flow goes to reference 1110, where the slice headerinformation of the slice and the data of the slice are transmitted tothe external storage separately upon obtaining a permission.

Transmitting the slice header information along with the encoded data ofthe slice through the one or more bus connections is advantageous. Theone or more bus connections may be implemented as parallel bus linesthat concurrently transmit multiple bits. Thus, the slice headerinformation may be transmitted ahead of the encoded data and within oneor more consecutive clock cycles thus speed up the processing of theslice. Additionally, the parallel bus line transmission may use burst,where the slice header information and the encoded data may betransmitted during a burst based on a single address.

When the one or more bus connections are implemented as parallel buslines that concurrently transmit multiple bits, the ideal transmissionis that each of the parallel bus lines has data to transmit. When someof the parallel bus lines have no data to transmit, dummy bits/bytes areadded for the bus lines to transmit. The dummy bits/bytes may be one ofall ones, all zeros, and other predetermined bit/byte patterns. Forexample, the one or more bus connections includes parallel bus lines forconcurrently transmitting 128 bits, and when the data in the buffer(e.g., the video data buffer 706) is insufficient to fulfil the parallelbus lines, dummy bits/bytes are transmitted along with the valid data.

The amount of data in the second memory location is generally less thanthe ones in the buffer, as the size of slice information is generallysmaller (e.g., the slice length takes two bytes in the example of FIG.9, thus the size of a slice index table is much smaller comparing to theencoded data that can be hundreds or thousands of bytes). In theexternal storage, a region is generally reserved to store the sliceinformation of a frame. With the parallel bus lines, it is desirable totransmit the slice information during a burst based on the singleaddress point to the region. Thus, a number of dummy bits/bytes may beadded for such burst transmission.

FIG. 12 is a flow diagram illustrating dummy byte insertion intransmission of slice information to an external storage according toone embodiment of the disclosure. Method 1200 may be implemented in anelectronic device such as the electronic device 150.

At reference 1202, the electronic device determines that the byte lengthof the slice information of a frame is less than the size of a regionreserved for the slice information of the slice. The region is onewithin an external storage such as the external storage 210.

At reference 1204, the electronic device calculates the number of dummybytes needed to fulfill the range reserved for the slice information.Then at reference 1206, the electronic device transmits the sliceinformation, an indication of the number of dummy bytes, and dummy bytesto the external storage through one or more bus connections.

The number of dummy bytes may be zero, or an integer number that withinthe byte width of the one or more bus connections in one embodiment. Forexample, assuming the parallel bus line is 128 bites thus 16 bytes, thenumber of dummy bytes may be a number between 0 and 14, so that one bytemay be used to indicate the number of the dummy bytes, and the rest ofthe bytes may be dummy bytes.

In an alternative embodiment, the number of dummy bytes may be betweenzero and an integer number that within the byte size of the region. Thusmultiple clock cycles may be used to transmit dummy bytes in suchembodiment. While dummy bytes are discussed for insertion, in analternative embodiment, dummy bits are used as the unit to calculate thenumber of dummy information to insert and to transmit to the externalstorage.

FIG. 13 illustrates memory allocation of an external storage accordingto one embodiment of the disclosure. The external storage may be theexternal storage 210. The data stored in the memory location is for onevideo frame. The width of the portion of storage for the video frame isthe byte length of the parallel bus to the external storage asillustrated at reference 1301. In an alternative embodiment, differentwidth of the portion of storage may be used.

The data is stored with the most significant bit (MSB) on the left andthe least significant bit (LSB) on the right. In this embodiment, thefirst byte is a dummy byte length indicator 1302 that indicates thelength of the dummy bytes. In an alternative embodiment, more bytes maybe used to indicate the length of the dummy bytes. Following the dummybyte length indictor is consecutive dummy bytes 1352 as indicated by thedummy length indicator. Following the dummy bytes are the sliceinformation, which includes slice index table.

As illustrated, the portion of storage starts with the region for sliceinformation 1304. The region includes a slice index table of the videoframe. The dummy byte length indicator may be read by a video processdriver, which then inserts information such as a frame header and/or apicture header of the video frame in the place of the dummy bytes. Whileless than a row of dummy bytes is illustrated, the dummy byte may takemore than one row in region for the slice information.

The region for the slice information is followed by slice header 1 atreference 1306, and slice data 1 at reference 1308, which include headerand data of slice 1 of the video frame. The following data of the videoframe is then stored in the order of a slice header being followed by aslice data of the same slice until all data of the video frame isstored.

All the data of the video frame stored in the portion of the storage isthen transmitted to a decoder. The dummy byte length indicator and thedummy bytes are removed prior to the transmission to the decoder. In oneembodiment, all the remaining data of the video frame after encoding aresealed in one or more packets, which are then transmitted through acommunication network such as the communication network 190, and thendecoded at an electronic device such as the affiliated device 152 by avideo decoder such as the video decoder 112 for display (e.g., by thedisplay 114) or further process.

FIG. 14 is an exemplary illustration of an electronic device forencoding video, in accordance with various embodiments of the presentdisclosure. The electronic device 1400 including many differentcomponents. These components can be implemented as integrated circuits(ICs), portions thereof, discrete electronic devices, or other modulesadapted to a circuit board such as a motherboard or add-in card of acomputing system, or as components otherwise incorporated within achassis of the computing system. Note also that the electronic device1400 is intended to show a high level view of many components of thecomputing system. However, it is to be understood that additionalcomponents may be present in certain implementations and furthermore,different arrangement of the components shown may occur in otherimplementations. In one embodiment, the electronic device is a movableobject as discussed herein above.

In one embodiment, the electronic device 1400 includes one or moremicroprocessors 1401, a video encoder 102, and non-transitorymachine-readable storage medium 1402, and optional devices 1403-1408that are interconnected via a bus or an interconnect 1410. The one ormore microprocessor 1401 represent one or more general-purposemicroprocessors such as a central processing unit (CPU), or processingdevice. More particularly, the microprocessor 1401 may be a complexinstruction set computing (CISC) microprocessor, reduced instruction setcomputing (RISC) microprocessor, very long instruction word (VLIW)microprocessor, or microprocessor implementing other instruction sets,or microprocessors implementing a combination of instruction sets.Microprocessor 1401 may also be one or more special-purpose processorssuch as an application specific integrated circuit (ASIC), a cellular orbaseband processor, a field programmable gate array (FPGA), a digitalsignal processor (DSP), a network processor, a graphics processor, anetwork processor, a communications processor, a cryptographicprocessor, a co-processor, an embedded processor, or any other type oflogic capable of processing instructions.

The one or more microprocessor 1401 may communicate with non-transitorymachine-readable storage medium 1402 (also called computer-readablestorage medium), such as magnetic disks, optical disks, read only memory(ROM), flash memory devices, and phase change memory. The non-transitorymachine-readable storage medium 1402 may store information includingsequences of instructions, such as computer programs, that are executedby the one or more microprocessors 1401, or any other device units. Forexample, executable code and/or data of a variety of operating systems,device drivers, firmware (e.g., input output basic system or BIOS),and/or applications can be loaded in the one or more microprocessor 1401and executed by the one or more microprocessor 1401.

The non-transitory machine-readable storage medium 1402 may store avideo process driver, which contains instructions and/or information toperform operations for video encoding such as identifying picture and/orframe headers of video frames in a video stream. Additionally, thenon-transitory machine-readable storage medium 1402 may include anexternal storage such as the external storage 210.

The video encoder 102 of the electronic device 1400 containsinstructions and/or information to perform operations for intra-slicevideo encoding as discussed herein above. The video encoder 102 may beimplemented using one or more circuits.

The optional propulsion unit 1403 may include one or more devices orsystems operable to generate forces for sustaining controlled movementof the electronic device 1400. The propulsion unit 1403 may share or mayeach separately include or be operatively connected to a power source,such as a motor (e.g., an electric motor, hydraulic motor, pneumaticmotor, etc.), an engine (e.g., an internal combustion engine, a turbineengine, etc.), a battery bank, etc., or combinations thereof. Thepropulsion unit 1403 may also include one or more rotary componentsconnected to the power source and configured to participate in thegeneration of forces for sustaining controlled flight. For instance,rotary components may include rotors, propellers, blades, nozzles, etc.,which may be driven on or by a shaft, axle, wheel, hydraulic system,pneumatic system, or other component or system configured to transferpower from the power source. The propulsion unit 1403 and/or rotarycomponents may be adjustable with respect to each other and/or withrespect to the electronic device 1400. The propulsion unit 1403 may beconfigured to propel the electronic device 1400 in one or more verticaland horizontal directions and to allow the electronic device 1400 torotate about one or more axes. That is, the propulsion unit 1403 may beconfigured to provide lift and/or thrust for creating and maintainingtranslational and rotational movements of the electronic device 1400.

The electronic device 1400 may optionally further include displaycontrol and/or display device unit 1404, wireless transceiver(s) 1405,video I/O device unit(s) 1406, audio I/O device unit(s) 1407, and otherI/O device units 1408 as illustrated. The wireless transceiver 1405 maybe a WiFi transceiver, an infrared transceiver, a Bluetooth transceiver,a WiMax transceiver, a wireless cellular telephony transceiver, asatellite transceiver (e.g., a global positioning system (GPS)transceiver), or other radio frequency (RF) transceivers, or acombination thereof.

The video I/O device unit 1406 may include an imaging processingsubsystem (e.g., a camera), which may include an optical sensor, such asa charged coupled device (CCD) or a complementary metal-oxidesemiconductor (CMOS) optical sensor, utilized to facilitate camerafunctions, such as recording photographs and video clips andconferencing. The video I/O device unit 1406 may be a 4K or 8Kcamera/camcorder in one embodiment.

An audio I/O device unit 1407 may include a speaker and/or a microphoneto facilitate voice-enabled functions, such as voice recognition, voicereplication, digital recording, and/or telephony functions. Otheroptional devices 1408 may include a storage device (e.g., a hard drive,a flash memory device), universal serial bus (USB) port(s), parallelport(s), serial port(s), a printer, a network interface, a bus bridge(e.g., a PCI-PCI bridge), sensor(s) (e.g., a motion sensor such as anaccelerometer, gyroscope, a magnetometer, a light sensor, compass, aproximity sensor, etc.), or a combination thereof. The optional deviceunits 1408 may further include certain sensors coupled to theinterconnect 1410 via a sensor hub (not shown), while other devices suchas a thermal sensor, an altitude sensor, an accelerometer, and anambient light sensor may be controlled by an embedded controller (notshown), dependent upon the specific configuration or design of theelectronic device 1400.

While various embodiments of the present disclosure have been describedabove, it should be understood that they have been presented by way ofexample, and not limitation. It will be apparent to persons skilled inthe relevant art that various changes in form and detail can be madetherein without departing from the spirit and scope of the disclosure.

The present disclosure has been described above with the aid offunctional building blocks illustrating the performance of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have often been arbitrarily defined herein for theconvenience of the description. Alternate boundaries can be defined solong as the specified functions and relationships thereof areappropriately performed. Any such alternate boundaries are thus withinthe scope and spirit of the disclosure.

The foregoing description of the present disclosure has been providedfor the purposes of illustration and description. It is not intended tobe exhaustive or to limit the disclosure to the precise forms disclosed.The breadth and scope of the present disclosure should not be limited byany of the above-described exemplary embodiments. Many modifications andvariations will be apparent to the practitioner skilled in the art. Themodifications and variations include any relevant combination of thedisclosed features. The embodiments were chosen and described in orderto best explain the principles of the disclosure and its practicalapplication, thereby enabling others skilled in the art to understandthe disclosure for various embodiments and with various modificationsthat are suited to the particular use contemplated. It is intended thatthe scope of the invention be defined by the following claims and theirequivalence.

What is claimed is:
 1. An electronic device comprising: a slice splitter configured to split video data into a plurality of slices, wherein each slice contains a plurality of data blocks and each data block contains a plurality of data points; and a slice encoder including one or more video encoding circuits and configured to: encode the data blocks in a plurality of data streams concurrently to obtain encoded data streams; and combine the encoded data streams into a combined data stream.
 2. The electronic device of claim 1, further comprising: a frequency transformer configured to transform the data blocks into a plurality of frequency coefficients.
 3. The electronic device of claim 2, wherein the frequency transformer is configured to perform a discrete cosine transform (DCT) on the data blocks.
 4. The electronic device of claim 2, wherein the plurality of frequency coefficients transformed from the plurality of data blocks within one slice are scanned in a frequency-first order or a block-first order.
 5. The electronic device of claim 2, further comprising: a quantizer configured to quantize the plurality of frequency coefficients from the frequency transformer to obtain quantized frequency coefficients; and a rate controller configured to control the quantization by the quantizer based on at least information from the slice encoder.
 6. The electronic device of claim 5, wherein the slice encoder is configured to encode the quantized frequency coefficients concurrently in the plurality of data streams.
 7. The electronic device of claim 2, wherein a combined encoding rate of the slice encoder that encodes the plurality of data streams concurrently matches a combined transforming rate of the frequency transformer in transforming the data blocks.
 8. The electronic device of claim 1, further comprising: a bit stream memory access circuit configured to cause the combined data stream to be stored in a buffer at a first memory location; and a header memory access circuit configured to cause slice information from the encoding to be stored in a second memory location relative to the first memory location.
 9. The electronic device of claim 8, wherein the slice information is contained in a slice index table.
 10. The electronic device of claim 8, wherein the bit stream memory access circuit includes a slice header formation sub-circuit configured to identify slice header information of the slices, the slice header information of one slice including at least one of: a slice header size; a scale factor; a size of luma data in the one slice; or a size of chroma data in the one slice.
 11. The electronic device of claim 8, wherein: the bit stream memory access circuit and the header memory access circuit are coupled to an external storage via one or more bus connections; and one of the bit stream memory access circuit and the header memory access circuit is further configured to transmit data to the external storage upon obtaining a permission.
 12. The electronic device of claim 11, wherein the header memory access circuit is further configured to cause the external storage to be notified of a difference between a byte length of the slice information and a byte length of the one or more bus connections.
 13. The electronic device of claim 1, wherein the plurality of data blocks form macroblocks, and one macroblock includes one or more data blocks from each of a luminance (Y) channel and two chrominance (Cr and Cb) channels.
 14. A method for encoding video data, comprising: splitting video data into a plurality of slices, wherein each slice contains a plurality of data blocks and each data block contains a plurality of data points; encoding the data blocks, through one or more video encoding circuits, in a plurality of data streams concurrently to obtain encoded data streams; and combining the encoded data streams into a combined data stream.
 15. The method of claim 14, further comprising: performing frequency transform on the data blocks to transform the data blocks into a plurality of frequency coefficients.
 16. The method of claim 15, wherein the plurality of frequency coefficients transformed from the plurality of data blocks within one slice are scanned in a frequency-first order or a block-first order.
 17. The method of claim 15, further comprising: quantizing the plurality of frequency coefficients under a control based on information from the one or more video encoding circuits.
 18. The method of claim 15, wherein a combined encoding rate to encode the plurality of data streams concurrently matches a combined transforming rate of the frequency transform of the data blocks.
 19. The method of claim 14, further comprising: storing the combined data stream in a buffer at a first memory location; and storing slice information from the encoding in a second memory location relative to the first memory location.
 20. An unmanned aircraft, comprising: a propulsion unit configured to effect a movement of the unmanned aircraft; a camera configured to generate video data; a slice splitter configured to split the video data into a plurality of slices, wherein each slice contains a plurality of data blocks and each data block contains a plurality of data points; and a slice encoder including one or more video encoding circuits and configured to: encode the data blocks in a plurality of data streams concurrently to obtain encoded data streams; and combine the encoded data streams into a combined data streams. 